{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scoring model performance with the `solaris` python API\n",
    "\n",
    "This tutorial describes how to run evaluation of a proposal (CSV or .geojson) for a single chip against a ground truth (CSV or .geojson) for the same chip.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CSV Eval\n",
    "\n",
    "### Steps\n",
    "1. Imports\n",
    "2. Load ground truth CSV\n",
    "3. Load proposal CSV\n",
    "4. Perform evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Imports  \n",
    "\n",
    "For this test case we will use the `eval` submodule within `solaris`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports\n",
    "import os\n",
    "import solaris as sol\n",
    "from solaris.data import data_dir\n",
    "import pandas as pd  # just for visualizing the outputs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "\n",
    "### Load ground truth CSV\n",
    "\n",
    "We will first instantiate an `Evaluator()` object, which is the core class `eval` uses for comparing predicted labels to ground truth labels. `Evaluator()` takes one argument - the path to the CSV or .geojson ground truth label object. It can alternatively accept a pre-loaded `GeoDataFrame` of ground truth label geometries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Evaluator sample_truth.csv"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ground_truth_path = os.path.join(data_dir, 'sample_truth.csv')\n",
    "\n",
    "evaluator = sol.eval.base.Evaluator(ground_truth_path)\n",
    "evaluator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, `evaluator` has the following attributes:\n",
    "\n",
    "- `ground_truth_fname`: the filename corresponding to the ground truth data. This is simply `'GeoDataFrame'` if a GDF was passed during instantiation.\n",
    "\n",
    "- `ground_truth_GDF`: GeoDataFrame-formatted geometries for the ground truth polygon labels.\n",
    "\n",
    "- `ground_truth_GDF_Edit`: A deep copy of `eval_object.ground_truth_GDF` which is edited during the process of matching ground truth label polygons to proposals.\n",
    "\n",
    "- `ground_truth_sindex`: The RTree/libspatialindex spatial index for rapid spatial referencing.\n",
    "\n",
    "- `proposal_GDF`: An _empty_ GeoDataFrame instantiated to hold proposals later.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Load proposal CSV\n",
    "\n",
    "Next we will load in the proposal CSV file. Note that the `proposalCSV` flag must be set to true for CSV data. If the CSV contains confidence column(s) that indicate confidence in proprosals, the name(s) of the column(s) should be passed as a list of strings with the `conf_field_list` argument; because no such column exists in this case, we will simply pass `conf_field_list=[]`. There are additional arguments available (see [the method documentation](https://cw-eval.readthedocs.io/en/latest/api.html#cw_eval.baseeval.EvalBase.load_proposal)) which can be used for multi-class problems; those will be covered in another recipe. The defaults suffice for single-class problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "proposals_path = os.path.join(data_dir, 'sample_preds.csv')\n",
    "evaluator.load_proposal(proposals_path, proposalCSV=True, conf_field_list=[])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "\n",
    "### Perform evaluation\n",
    "\n",
    "Evaluation iteratively steps through the proposal polygons in `eval_object.proposal_GDF` and determines if any of the polygons in `eval_object.ground_truth_GDF_Edit` have IoU overlap > `miniou` (see [the method documentation](https://cw-eval.readthedocs.io/en/latest/api.html#cw_eval.baseeval.EvalBase.eval_iou)) with that proposed polygon. If one does, that proposal polygon is scored as a true positive. The matched ground truth polygon with the highest IoU (in case multiple had IoU > `miniou`) is removed from `eval_object.ground_truth_GDF_Edit` so it cannot be matched against another proposal. If no ground truth polygon matches with IoU > `miniou`, that proposal polygon is scored as a false positive. After iterating through all proposal polygons, any remaining ground truth polygons in `eval_object.ground_truth_GDF_Edit` are scored as false negatives.\n",
    "\n",
    "There are several additional arguments to this method related to multi-class evaluation which will be covered in a later recipe. See [the method documentation](https://cw-eval.readthedocs.io/en/latest/api.html#cw_eval.baseeval.EvalBase.eval_iou) for usage.\n",
    "\n",
    "The prediction outputs a `list` of `dict`s for each class evaluated (only one `dict` in this single-class case). The `dict`(s) have the following keys:\n",
    "\n",
    "- `'class_id'`: The class being scored in the dict, `'all'` for single-class scoring.\n",
    "\n",
    "- `'iou_field'`: The name of the column in `eval_object.proposal_GDF` for the IoU score for this class. See [the method documentation](https://cw-eval.readthedocs.io/en/latest/api.html#cw_eval.baseeval.EvalBase.eval_iou) for more information.\n",
    "\n",
    "- `'TruePos'`: The number of polygons in `eval_object.proposal_GDF` that matched a polygon in `eval_object.ground_truth_GDF_Edit`.\n",
    "\n",
    "- `'FalsePos'`: The number of polygons in `eval_object.proposal_GDF` that had no match in `eval_object.ground_truth_GDF_Edit`.\n",
    "\n",
    "- `'FalseNeg'`: The number of polygons in `eval_object.ground_truth_GDF_Edit` that had no match in `eval_object.proposal_GDF`.\n",
    "\n",
    "- `'Precision'`: The [precision statistic](https://en.wikipedia.org/wiki/Precision_and_recall) for IoU between the proposals and the ground truth polygons.\n",
    "\n",
    "- `'Recall'`: The [recall statistic](https://en.wikipedia.org/wiki/Precision_and_recall) for IoU between the proposals and the ground truth polygons.\n",
    "\n",
    "- `'F1Score'`: Also known as the [SpaceNet Metric](https://medium.com/the-downlinq/the-spacenet-metric-612183cc2ddb), the [F<sub>1</sub> score](https://en.wikipedia.org/wiki/F1_score) for IoU between the proposals and the ground truth polygons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "151it [00:00, 153.44it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'class_id': 'all',\n",
       "  'iou_field': 'iou_score_all',\n",
       "  'TruePos': 151,\n",
       "  'FalsePos': 0,\n",
       "  'FalseNeg': 0,\n",
       "  'Precision': 1.0,\n",
       "  'Recall': 1.0,\n",
       "  'F1Score': 1.0}]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluator.eval_iou(calculate_class_scores=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the score is perfect because the polygons in the ground truth CSV and the proposal CSV are identical. At this point, a new proposal CSV can be loaded (for example, for a new nadir angle at the same chip location) and scoring can be repeated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## GeoJSON Eval\n",
    "\n",
    "The same operation can be completed with .geojson-formatted ground truth and proposal files. See the example below, and see the detailed explanation above for a description of each step's operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "28it [00:00, 76.20it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'class_id': 'all',\n",
       "  'iou_field': 'iou_score_all',\n",
       "  'TruePos': 8,\n",
       "  'FalsePos': 20,\n",
       "  'FalseNeg': 20,\n",
       "  'Precision': 0.2857142857142857,\n",
       "  'Recall': 0.2857142857142857,\n",
       "  'F1Score': 0.2857142857142857}]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ground_truth_geojson = os.path.join(data_dir, 'gt.geojson')\n",
    "proposal_geojson = os.path.join(data_dir, 'pred.geojson')\n",
    "\n",
    "evaluator = sol.eval.base.Evaluator(ground_truth_geojson)\n",
    "evaluator.load_proposal(proposal_geojson, proposalCSV=False, conf_field_list=[])\n",
    "evaluator.eval_iou(calculate_class_scores=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Note that the above comes from a different chip location and different proposal than the CSV example, hence the difference in scores)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "solaris",
   "language": "python",
   "name": "solaris"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
